Synchronizing garbage collection and incoming data traffic

ABSTRACT

The technology describes performing garbage collection while data writes are occurring, which can lead to a conflict in that a new reference to an otherwise non-referenced candidate object for garbage collection is written after the non-referenced candidate object is detected. In one example implementation, orphaned binary large objects (BLOBs) that are not referenced by a descriptor file and are beyond a certain age are detected and deleted via an object references table traversal as part of garbage collection. Before reclaiming a deleted BLOB&#39;s capacity, a background process operates to restore the deleted BLOB if a new descriptor file reference to the BLOB was written during the object references table traversal. Capacity is only reclaimed after the object references table traversal and the background processing completes, for those BLOBs that were deleted and had not been restored.

TECHNICAL FIELD

The subject application generally relates to data storage, and, forexample, to a data storage system that facilitates performing garbagecollection without data loss and without the need to halt the incomingdata traffic, and related embodiments.

BACKGROUND

Contemporary cloud-based data storage systems, such as ECS (formerlyknown as ELASTIC CLOUD STORAGE) provided by DELL EMC, support datasystems that are based on content addressable storage, such as CENTERA.In CENTERA, a data object received from an application is stored as aBLOB (Binary Large Object) and stored at a content address calculatedbased on the object's content. The address and object metadata areinserted into an XML file referred to as a C-Clip descriptor file (CDF),which in turn has its content address calculated. This C-Clip's addressis returned to the application once the CDF and BLOB have beensuccessfully stored and protected in the storage. From the ECS point ofview, BLOBs and CDFs are fully independent objects. There is a thin CASimplementation layer that connects BLOBs and CDFs in order to serve datawrites and reads

BLOB deletion works at the CDF level, and when a BLOB no longer has anyCDF references to it, then that BLOB is referred to as an orphaned BLOB.Such a BLOB object can be deleted and a garbage collection engine in ECScan reclaim the capacity the BLOB occupied. It is possible to have aBLOB become an orphan, yet become referenced by a CDF again as a resultof a new CDF write. Note that ECS supports geographically distributedsetups of two or more zones, where each zone is normally an ECS cluster;this makes the scenario of an orphaned BLOB again becoming referenced bya CDF even more probable.

In order to prevent a possible data loss event, when a CDF references anon-existent BLOB, each BLOB referenced by a new CDF has to be updatedwith a new reference to it in a foreground operation of the CDF writetransaction. This prevents deletion of the BLOBs; that is, thereferences need to be added before client acknowledgement for the newCDF is sent. This serves as a kind of write barrier that severelyimpacts storage performance. Consider an example in which there are twoECS zone, in which a new CDF (which references one the order of 100,000BLOBs) is written in the first zone write, with these BLOBs owned by thesecond (remote) zone. From the client perspective, creation of such aCDF, which is only of 10-20 MB size, will take an extremely long timebecause the client will not get an acknowledgement until the first zoneupdates system metadata of the 100,000 remotely owned BLOBs. Onesolution is to prevent incoming data traffic in a “stop-the-world”disruptive mode in which there is no need for synchronization withgarbage collection that deletes orphaned BLOBS because there is noincoming data traffic; however this is a very undesirable solution.

BRIEF DESCRIPTION OF THE DRAWINGS

The technology described herein is illustrated by way of example and notlimited in the accompanying figures in which like reference numeralsindicate similar elements and in which:

FIG. 1 is an example block diagram representation of part of a datastorage system including nodes, in which garbage collection and datawrites can occur without data loss via snapshot restoration, inaccordance with various aspects and implementations of the subjectdisclosure.

FIG. 2 is an example block diagram representation of part of a datastorage system that performs garbage collection while new data writesare allowed to take place, with deleted object data restored fromsnapshot data in the event a new data write references a deleted object,in accordance with various aspects and implementations of the subjectdisclosure.

FIG. 3 is a flow diagram representing example operations related todetermining orphaned data objects for deletion via garbage collection ascandidates for having their capacity reclaimed, in accordance withvarious aspects and implementations of the subject disclosure.

FIG. 4 is a flow diagram representing example operations related towaiting for deleted data objects to possibly be restored via abackground process, and for reclaiming capacity of deleted data objectsthat have not been restored, in accordance with various aspects andimplementations of the subject disclosure.

FIG. 5 is a flow diagram representing example operations related tohandling a new descriptor file write via a foreground operation, inaccordance with various aspects and implementations of the subjectdisclosure.

FIG. 6 is a flow diagram representing example operations related tohandling a new descriptor file write via a background operation,including to restore a deleted data object that is referenced by a newdescriptor file written in the foreground operations of FIG. 5, inaccordance with various aspects and implementations of the subjectdisclosure.

FIG. 7 is a flow diagram showing example operations related to detectingdata objects for garbage collection if not restored, in accordance withvarious aspects and implementations of the subject disclosure.

FIG. 8 is a flow diagram showing example operations related to usingsnapshot data as needed to restore data objects, and reclaiming capacityvia garbage collection of data objects not restored in accordance withvarious aspects and implementations of the subject disclosure.

FIG. 9 is a flow diagram showing example operations related to creatingsnapshot data, deleting non-referenced and older objects in a firstgarbage collection process, and reclaiming capacity via a second garbagecollection process those data objects that are deleted and not restoredvia the snapshot data, in accordance with various aspects andimplementations of the subject disclosure.

FIG. 10 depicts an example schematic block diagram of a computingenvironment with which the disclosed subject matter can interact, inaccordance with various aspects and implementations of the subjectdisclosure.

FIG. 11 illustrates an example block diagram of a computing systemoperable to execute the disclosed systems and methods in accordance withvarious aspects and implementations of the subject disclosure.

DETAILED DESCRIPTION

Various aspects of the technology described herein are generallydirected towards synchronize garbage collection and incoming datatraffic in a manner that avoids a data loss event yet without the needto halt incoming data traffic from a mutator (e.g. application program).In one aspect, snapshots are used to eliminate write barriers, where asnapshot comprises an available, near instantaneous back-up copy of datacreated at a particular point in time. Note that in ECS, snapshots canbe considered at the “bucket” level or simply bucket snapshots withrespect to buckets with content addressable storage (CAS) accessenabled.

In general and as described herein, a CAS garbage collection processtraverses an object reference table (a search tree) to identify binarylarge objects (BLOBs) that are orphans, and deletes them; note howeverthat such deletion does not reclaim their capacity. During the objectreferences table traversal/CAS garbage collection operations, it ispossible that a new C-Clip Descriptor File (CDF) is written (in aforeground process) that references a deleted orphan BLOB. If so, (in abackground process), the BLOB is restored from an appropriate snapshot.

After the object references table traversal is finished, and when thebackground process completes for any CDF written during the objectreferences table traversal, the snapshots are deleted by the CAS garbagecollection process. Further an ECS garbage collection engine (a nativegarbage collection process implemented in an ECS Engine) operates toreclaim capacity of the deleted BLOBs that have not been restored.

The technology described herein, in which possible conflicts aredetected in the background mode and resolved by means of snapshots, thussolves the problem of synchronization between garbage collection andincoming data traffic in a cloud storage. Indeed, from one perspectivethe technology can be considered as eliminating the need to do anysynchronization, with only a relatively short delay in actual capacityreclamation for deleted orphan BLOBs.

As will be understood, the implementation(s) described herein arenon-limiting examples, and variations to the technology can beimplemented. For example, in ECS cloud storage technology the objectreferences table is stored in a search tree data structure (a directorytable), however any data storage or other system that needs to performgarbage collection without halting incoming data traffic may benefitfrom the present technology. Indeed, it should be understood that any ofthe examples herein are non-limiting. For instance, some of the examplesare based on ECS cloud storage technology; however virtually any storagesystem may benefit from the technology described herein. Further, manyof the examples refer to CAS binary large objects and CDF files, howeverattentive data storage systems with other data and metadata structuresmay be used with the technology described herein. Still further,snapshots are described as a convenient way to temporarily preserve datafor restoring if needed, however any technology that preserves data canbe used. Thus, any of the embodiments, aspects, concepts, structures,functionalities or examples described herein are non-limiting, and thetechnology may be used in various ways that provide benefits andadvantages in computing and data storage in general.

FIG. 1 shows part of a cloud data storage system such as ECS comprisinga zone (e.g., cluster) 102 of storage nodes 104(1)-104(N), in which eachnode is typically a server configured primarily to serve objects inresponse to client requests. The nodes 104(1)-104(N) are coupled to eachother via a suitable data communications link comprising interfaces andprotocols, such as represented in FIG. 1 by Ethernet block 106.

Clients 108 make data system-related requests to the cluster 102, whichin general is configured as one large object namespace; there may be onthe order of billions of objects maintained in a cluster, for example.To this end, a node such as the node 104(2) generally comprises ports112 by which clients connect to the cloud storage system. Example portsare provided for requests via various protocols, including but notlimited to SMB (server message block), FTP (file transfer protocol),HTTP/HTTPS (hypertext transfer protocol) and NFS (Network File System);further, SSH (secure shell) allows administration-related requests, forexample.

In general, and in one or more implementations, e.g., ECS, disk space ispartitioned into a set of relatively large blocks of typically fixedsize (e.g., 128 MB) referred to as chunks; user data is generally storedin chunks, e.g., in a user data repository. Normally, one chunk containssegments of several user objects. In other words, chunks can be shared,that is, one chunk may contain segments of multiple user objects; e.g.,one chunk may contain mixed segments of some number of (e.g., three)user objects. Tree data can also be maintained in tree chunks.

In FIG. 1, each node, such as the node 104(2), includes an instance of adata storage system 114 and data services, including geographic (geo)data services 116 that facilitate coupling to other geographic zones118, such as for replicating data to the other zones' data storage 120.Note however that at least some data service components can beper-cluster, or per group of nodes, rather than per-node.

ECS runs a set of storage services, which together implement storagebusiness logic. Services can also maintain directory tables for keepingtheir metadata, which can be implemented as search trees; the data ofsearch trees can be kept in tree chunk data structures. For example, anobject table (one type of directory table) keeps track of objects in thedata storage system and generally stores the system objects' metadata,including an object's data location within a chunk data structure. Notethat the object table can be partitioned among the nodes 104(1)-104(N)of the cluster. There is also a “reverse” directory table (maintained byanother service) that keeps a per chunk list of objects that have theirdata in a particular chunk.

In the example of FIG. 1, each node, such as the node 104(2), caninclude content addressable storage (CAS) components 122, includingbinary large objects (BLOBs) and descriptor files (C-Clip descriptorfiles, or CDFs). The CAS components also include a garbage collectionprocess that works at the distributed storage level; in one or moreimplementations, the CAS garbage collection process creates snapshots124 for the CAS-enabled buckets as described herein with reference toFIG. 2.

As set forth above, from the ECS point of view, such as represented inFIG. 1 by ESC components 126, BLOBs and CDFs are fully independentobjects, and there is a CAS implementation layer that connects BLOBs andCDFs in order to serve data writes and reads. The ESC components 126also include a garbage collection engine, as described herein withreference to FIG. 2, that reclaims capacity of deleted, orphaned BLOBs

In FIG. 1, a CPU 130 and RAM 132 are shown; note that the RAM 132 maycomprise at least some non-volatile RAM. The node includes storagedevices such as disks 134, comprising hard disk drives and/orsolid-state drives. As is understood, any node data structure such as anobject, object table, chunk table, chunk, code, and the like can be inRAM 128, on disk(s) 130 or a combination of partially in RAM, partiallyon disk, backed on disk, replicated to other nodes and so on.

FIG. 2 shows example components including data that facilitate aspectsof the technology described herein. In FIG. 2, data writes 222 result inCAS BLOB data 226(a) being created and deleted, along with descriptorfiles that reference the BLOBs. In general, a client application/programor the like delivers a data object, from which the content address (CA)is calculated and stored along with metadata about the object into aC-Clip Descriptor File (CDF) in CDF data storage 228. One CDF canreference up to 100,000 BLOBs, and one BLOB can be referenced bymillions of CDFs. The BLOB(s) are stored in the storage before thereferencing CDF, e.g., in a transaction, which if successful results inan acknowledgment (ACK) 230 being returned to the writer/clientapplication.

BLOB deletion operates at the CDF level, and once the CDF references toa BLOB have been deleted, the data objects (BLOBs) become orphans. Thecapacity occupied by the orphans needs to be reclaimed via garbagecollection. However, detecting orphans in a reliable way is notstraightforward, because BLOBs can gain and lose references to themasynchronously, and because of the cluster environment, references to aBLOB can be handled independently on different cluster nodes.

In FIG. 2, when orphans are to be detected and deleted as describedherein, a CAS garbage collection process 232 creates instant snapshots224 for any buckets having CAS access enabled. Once the snapshots 224are created, the CAS garbage collection process 232 traverses an objectreferences (directory) table 234, which in one implementation is asearch tree, to obtain information for the BLOBs. Note that the objectreferences directory table tracks BLOB references, in which the key is ablob content address (CA), and the value is the set of content addressesof the CDFs that reference the blob.

In general, the CAS garbage collection process 232 looks for orphanedBLOBS (block 236), and deletes those deemed true orphans (block 238).More particularly, as the object reference table is traversed, for eachBLOB, the CAS garbage collection process 232 checks if the BLOB is anorphan, that is, if there are no CDF references to that BLOB. If theBLOB is not an orphan, the CAS garbage collection process 232 skipsfurther processing of the BLOB. If the BLOB is an orphan, the CASgarbage collection process 232 checks the BLOB age; if the BLOB isyounger than the maximum duration of a C-Clip write transaction (twoweeks in one implementation), the CAS garbage collection process 232skips further processing of the BLOB.

However, if the BLOB being evaluated is two weeks old or older, the BLOBis considered to be a true orphan, and the CAS garbage collectionprocess 232 deletes the BLOB. Note that this only makes the deleted BLOBa “candidate” for actual garbage collection, as this deletion does notreclaim the capacity of the BLOB, which may be done later if appropriate(that is, not restored) as described herein. Traversal continues toevaluate the next BLOB and so on until the traversal is finished.

After traversal of the object references table 234 is finished, the CASgarbage collection process 232 waits until background processing (block240) of any CDFs that were written (by a foreground process, block 242)during object references table traversal, has completed. Significantly,the background process 240 can restore BLOBs (block 244) that wereconsidered orphaned and deleted, but became referenced by a new CDFwritten during the object references table traversal, which causes aconflicting state. Once such background processing has completed, theCAS garbage collection process 232 deletes the snapshots 224 that itcreated at the beginning.

After both the object references table 234 has been traversed and thebackground processing (block 240) has completed, an ECS garbagecollection engine 246 reclaims capacity (block 248) occupied by the BLOBobjects that remain deleted. The result is an updated set of BLOB data226(b), which can contain one or more orphaned BLOBs (block 250) thatare less than the age of a maximum CDF transaction.

FIG. 3 show the general example logic of the CAS garbage collectionprocess 232, beginning at operation 302 where the bucket snapshots arecreated to facilitate the background (or delayed) processing of BLOBreferences in new CDFs as described herein.

Operation 304 represents the CAS garbage collection process 232traversing the object references table to locate a BLOB reference. Viaoperation 306, for each BLOB found, at operation 308 the CAS garbagecollection process 232 checks whether the BLOB is an orphan, that is,without CDF references to it. If the BLOB is not an orphan, operation308 skips further processing of the BLOB by branching to operation 316,which continues the object references table traversal until done.

If instead at operation 308 the BLOB is an orphan, the CAS garbagecollection process 232 determines the BLOB age, which is contained inobject metadata for the BLOB. If at operation 312 the BLOB is youngerthan the (maximum duration of a CDF write transaction (e.g., two weeks),operation 312 skips further processing of the BLOB. Conversely, if atoperation 312 the BLOB is at or is older than the maximum duration of aCDF write transaction, the CAS garbage collection process 232 considersthe BLOB to be a true orphan, and at operation 314 deletes the BLOB.Traversal continues via operation 316 until the object references tableis fully traversed.

FIG. 4 shows waiting, after traversal of the object references table, atoperation 402 until the background CDF processing (FIG. 6) completes,e.g., as signaled to the data storage system at the end of backgroundCDF processing. When background processing, which handles CDFs that werewritten during object references traversal has completed, the CASgarbage collection process 232 deletes the previously created bucketsnapshots (which may be used by the background CDF processing to restoredeleted BLOBs as described herein).

At this time, any BLOBs that remain deleted are orphans that did not getreferenced by a new CDF written during the object references tabletraversal. At operation 406, the native ESC garbage collection process(e.g., implemented in an ECS engine) reclaims capacity occupied by thedeleted BLOB objects.

FIG. 5 summaries how the data storage system handles a new CDF, in aforeground mode/process, corresponding to the CDF write foregroundprocessing 242 of FIG. 2. At operation 502 the system stores andprotects the content of the CDF. At operation 504, the data storagesystem checks whether the BLOBs referenced by the CDF exist. If at leastone BLOB referenced by the CDF does not exist, at operation 506 the datastorage system stem fails the entire CDF write transaction. Conversely,if the BLOB(s) referenced by the CDF exist, the data storage systemacknowledges the CDF creation locally, and sends an acknowledgementmessage to the client, as represented by operation 508. The foregroundmode then ends.

It should be noted that operation 504 of the foreground mode above thatchecks for a BLOB's existence is not part of the logic of the CASgarbage collection process, but rather follows from the general CENTERACAS implementation. Further note that the check for a BLOB's existenceis a very fast operation relative to a system metadata update operation.

FIG. 6 summaries how the data storage system handles one or more newCDFs, in a background mode/process corresponding to the CDF writebackground P=processing 240 of FIG. 2. In general, the data storagesystem tries to update each BLOB referenced by the CDF with a newreference to the BLOB.

At operation 602, the data storage system selects a new CDF (written inthe foreground mode during the object references traversal), and selectsa BLOB referenced by that CDF at operation 604. If the CDF-referencedBLOB does not exist, at operation 608 the BLOB restored from thecorresponding snapshot. The referenced BLOB also gets updated with a newreference to the CDF at operation 610. Note that this may happen onlywhile the CAS garbage collection process is active; the expectedprobability of this event is very low. The operations continue with thenext referenced BLOB and so on via operation 612 until no BLOBsreferenced by the CDF remain; when this occurs, the CDF is marked asprocessed at operation 614.

If another CDF has not yet been handled in the background mode,operation 616 repeats the process until no CDFs remain to be processed.When none remain, the background mode ends. At this time, the datastorage system knows whether the object references table traversal hascompleted, and if so, ends the wait at operation 402 of FIG. 4 to beginreclaiming capacity as described herein. Otherwise it is possible thatat least one new CDF may be written during further traversal of theobject references table, in which event the operations of FIGS. 4 and 5will be performed for the at least one new CDF.

One or more aspects are represented in FIG. 7, such as of a systemcomprising a processor, and a memory that stores executable instructionsthat, when executed by the processor, facilitate performance ofoperations. Operation 702 represents traversing a tree data structure todetect an orphaned content addressable storage object comprising acontent addressable storage object not referenced by a descriptor file.Operation 704 represents determining whether the orphaned contentaddressable storage object satisfies an age criterion. Operation 706represents, in response to determining that the detected orphanedcontent addressable storage object satisfies the age criterion, deletingthe content addressable storage object resulting in a deleted contentaddressable storage object. Operation 708 represents determining whetherany new descriptor file written during the traversing has restored thedeleted content addressable storage object. Operation 710 represents, inresponse to determining that the deleted content addressable storageobject has not been restored during the traversing, reclaiming capacityoccupied by the deleted content addressable storage object.

Further operations can comprise creating a snapshot of a bucketcontaining the content addressable storage object. Further operationscan comprise deleting the snapshot after determining whether the any newdescriptor file written during the traversing has restored the deletedcontent addressable storage object. The orphaned content addressablestorage object can be a first orphaned content addressable storageobject, and further operations can comprise creating a new descriptorfile, determining that a second orphaned content addressable storageobject referenced by the new descriptor file does not exist, andrestoring the second orphaned content addressable storage object fromthe snapshot.

The age criterion can be based on a maximum duration of a descriptorfile write transaction. The maximum duration can be two weeks.

Deleting the content addressable storage object can be performed by afirst garbage collection process, and reclaiming the capacity occupiedby the deleted content addressable storage object can be performed by asecond garbage collection engine.

The detected content addressable storage object can be a binary largeobject. The descriptor file can be a C-Clip descriptor file.

One or more aspects are represented in FIG. 8, such as exampleoperations of a method. Operation 802 represents creating, via aprocessor of a data storage system, a snapshot of a container containingcontent addressable storage objects. Operation 804 represents traversingan object references data structure corresponding to the container todetermine orphaned content addressable storage objects comprisingcontent addressable storage objects that are not referenced by at leastone descriptor file. Operation 806 represents deleting, from thecontainer, the orphaned content addressable storage objects that areolder than a predetermined descriptor file write duration, the deletingresulting in deleted content addressable storage objects. Operation 808represents reclaiming capacity occupied by the deleted contentaddressable storage objects that have not been restored from thesnapshot via any new descriptor file created during the traversing.

Traversing the object references data structure can comprise running afirst garbage collection operation, and wherein the reclaiming thecapacity comprises running a second garbage collection operation.

Aspects can comprise writing a new descriptor file, during thetraversing and before the reclaiming the capacity, that references adeleted content addressable storage object, and restoring the contentaddressable storage object from the snapshot to an undeleted state.

Writing the new descriptor file can comprise running a foregroundprocess, and restoring the content addressable storage object from thesnapshot to the undeleted state can comprise running a backgroundprocess. Reclaiming the capacity can be performed after the backgroundprocess completes.

Traversing the object references data structure can comprise traversinga search tree.

Creating the snapshot of the container can comprise creating a firstsnapshot of a first container, and aspects can comprise creatingrespective one or more snapshots of one or more respective containers.

One or more aspects, such as implemented in a machine-readable storagemedium, can comprise executable instructions that, when executed by aprocessor of a data storage system, can be directed towards operationsexemplified in FIG. 9. Example operation 902 represents creating asnapshot of a container containing content addressable storage objects.Example operation 904 represents deleting, as part of a first garbagecollection process, a content addressable storage object, maintained inthe container, which is not referenced by a descriptor file and is olderthan an age that is based on a defined limit on duration of a descriptorfile write transaction. Example operation 906 represents reclaimingcapacity, as part of a second garbage collection process, the contentaddressable storage object in response to determining that the contentaddressable storage object was unable to be restored from the snapshotduring the first garbage collection process.

The content addressable storage object can be a first contentaddressable storage object, and further operations can comprise,deleting, as part of the first garbage collection process, a secondcontent addressable storage object, maintained in the container, that isnot referenced by any descriptor file and is older than the age that isbased on the defined limit on the duration of a descriptor file writetransaction, writing a new descriptor file with a reference to thesecond content addressable storage object, restoring the second contentaddressable storage object from the snapshot, and avoiding reclamationof capacity of the second content addressable storage object during thesecond garbage collection process.

Writing the new descriptor file can comprise running a foregroundprocess, restoring the second content addressable storage object fromthe snapshot can comprise running a background process that starts afterthe foreground process completes, and the reclaiming the capacity in thesecond garbage collection process can start after the background processcompletes.

The operations can comprise traversing an object references datastructure as part of the first garbage collection process.

As can be seen, described herein is a technology that facilitatesperforming garbage collection without halting data writes and without adata loss event. The technology uses snapshots (or other preserved data)to restore data in a background mode when a possible conflict isdetected. The system is practical to implement.

FIG. 10 is a schematic block diagram of a computing environment 1000with which the disclosed subject matter can interact. The system 1000comprises one or more remote component(s) 1010. The remote component(s)1010 can be hardware and/or software (e.g., threads, processes,computing devices). In some embodiments, remote component(s) 1010 can bea distributed computer system, connected to a local automatic scalingcomponent and/or programs that use the resources of a distributedcomputer system, via communication framework 1040. Communicationframework 1040 can comprise wired network devices, wireless networkdevices, mobile devices, wearable devices, radio access network devices,gateway devices, femtocell devices, servers, etc.

The system 1000 also comprises one or more local component(s) 1020. Thelocal component(s) 1020 can be hardware and/or software (e.g., threads,processes, computing devices). In some embodiments, local component(s)1020 can comprise an automatic scaling component and/or programs thatcommunicate/use the remote resources 1010 and 1020, etc., connected to aremotely located distributed computing system via communicationframework 1040.

One possible communication between a remote component(s) 1010 and alocal component(s) 1020 can be in the form of a data packet adapted tobe transmitted between two or more computer processes. Another possiblecommunication between a remote component(s) 1010 and a localcomponent(s) 1020 can be in the form of circuit-switched data adapted tobe transmitted between two or more computer processes in radio timeslots. The system 1000 comprises a communication framework 1040 that canbe employed to facilitate communications between the remote component(s)1010 and the local component(s) 1020, and can comprise an air interface,e.g., Uu interface of a UMTS network, via a long-term evolution (LTE)network, etc. Remote component(s) 1010 can be operably connected to oneor more remote data store(s) 1050, such as a hard drive, solid statedrive, SIM card, device memory, etc., that can be employed to storeinformation on the remote component(s) 1010 side of communicationframework 1040. Similarly, local component(s) 1020 can be operablyconnected to one or more local data store(s) 1030, that can be employedto store information on the local component(s) 1020 side ofcommunication framework 1040.

In order to provide additional context for various embodiments describedherein, FIG. 11 and the following discussion are intended to provide abrief, general description of a suitable computing environment 1100 inwhich the various embodiments of the embodiment described herein can beimplemented. While the embodiments have been described above in thegeneral context of computer-executable instructions that can run on oneor more computers, those skilled in the art will recognize that theembodiments can be also implemented in combination with other programmodules and/or as a combination of hardware and software.

Generally, program modules include routines, programs, components, datastructures, etc., that perform particular tasks or implement particularabstract data types. Moreover, those skilled in the art will appreciatethat the methods can be practiced with other computer systemconfigurations, including single-processor or multiprocessor computersystems, minicomputers, mainframe computers, Internet of Things (IoT)devices, distributed computing systems, as well as personal computers,hand-held computing devices, microprocessor-based or programmableconsumer electronics, and the like, each of which can be operativelycoupled to one or more associated devices.

The illustrated embodiments of the embodiments herein can be alsopracticed in distributed computing environments where certain tasks areperformed by remote processing devices that are linked through acommunications network. In a distributed computing environment, programmodules can be located in both local and remote memory storage devices.

Computing devices typically include a variety of media, which caninclude computer-readable storage media, machine-readable storage media,and/or communications media, which two terms are used herein differentlyfrom one another as follows. Computer-readable storage media ormachine-readable storage media can be any available storage media thatcan be accessed by the computer and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable storage media or machine-readablestorage media can be implemented in connection with any method ortechnology for storage of information such as computer-readable ormachine-readable instructions, program modules, structured data orunstructured data.

Computer-readable storage media can include, but are not limited to,random access memory (RAM), read only memory (ROM), electricallyerasable programmable read only memory (EEPROM), flash memory or othermemory technology, compact disk read only memory (CD-ROM), digitalversatile disk (DVD), Blu-ray disc (BD) or other optical disk storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, solid state drives or other solid statestorage devices, or other tangible and/or non-transitory media which canbe used to store desired information. In this regard, the terms“tangible” or “non-transitory” herein as applied to storage, memory orcomputer-readable media, are to be understood to exclude onlypropagating transitory signals per se as modifiers and do not relinquishrights to all standard storage, memory or computer-readable media thatare not only propagating transitory signals per se.

Computer-readable storage media can be accessed by one or more local orremote computing devices, e.g., via access requests, queries or otherdata retrieval protocols, for a variety of operations with respect tothe information stored by the medium.

Communications media typically embody computer-readable instructions,data structures, program modules or other structured or unstructureddata in a data signal such as a modulated data signal, e.g., a carrierwave or other transport mechanism, and includes any information deliveryor transport media. The term “modulated data signal” or signals refersto a signal that has one or more of its characteristics set or changedin such a manner as to encode information in one or more signals. By wayof example, and not limitation, communication media include wired media,such as a wired network or direct-wired connection, and wireless mediasuch as acoustic, RF, infrared and other wireless media.

With reference again to FIG. 11, the example environment 1100 forimplementing various embodiments of the aspects described hereinincludes a computer 1102, the computer 1102 including a processing unit1104, a system memory 1106 and a system bus 1108. The system bus 1108couples system components including, but not limited to, the systemmemory 1106 to the processing unit 1104. The processing unit 1104 can beany of various commercially available processors. Dual microprocessorsand other multi-processor architectures can also be employed as theprocessing unit 1104.

The system bus 1108 can be any of several types of bus structure thatcan further interconnect to a memory bus (with or without a memorycontroller), a peripheral bus, and a local bus using any of a variety ofcommercially available bus architectures. The system memory 1106includes ROM 1110 and RAM 1112. A basic input/output system (BIOS) canbe stored in a non-volatile memory such as ROM, erasable programmableread only memory (EPROM), EEPROM, which BIOS contains the basic routinesthat help to transfer information between elements within the computer1102, such as during startup. The RAM 1112 can also include a high-speedRAM such as static RAM for caching data.

The computer 1102 further includes an internal hard disk drive (HDD)1114 (e.g., EIDE, SATA), and can include one or more external storagedevices 1116 (e.g., a magnetic floppy disk drive (FDD) 1116, a memorystick or flash drive reader, a memory card reader, etc.). While theinternal HDD 1114 is illustrated as located within the computer 1102,the internal HDD 1114 can also be configured for external use in asuitable chassis (not shown). Additionally, while not shown inenvironment 1100, a solid state drive (SSD) could be used in additionto, or in place of, an HDD 1114.

Other internal or external storage can include at least one otherstorage device 1120 with storage media 1122 (e.g., a solid state storagedevice, a nonvolatile memory device, and/or an optical disk drive thatcan read or write from removable media such as a CD-ROM disc, a DVD, aBD, etc.). The external storage 1116 can be facilitated by a networkvirtual machine. The HDD 1114, external storage device(s) 1116 andstorage device (e.g., drive) 1120 can be connected to the system bus1108 by an HDD interface 1124, an external storage interface 1126 and adrive interface 1128, respectively.

The drives and their associated computer-readable storage media providenonvolatile storage of data, data structures, computer-executableinstructions, and so forth. For the computer 1102, the drives andstorage media accommodate the storage of any data in a suitable digitalformat. Although the description of computer-readable storage mediaabove refers to respective types of storage devices, it should beappreciated by those skilled in the art that other types of storagemedia which are readable by a computer, whether presently existing ordeveloped in the future, could also be used in the example operatingenvironment, and further, that any such storage media can containcomputer-executable instructions for performing the methods describedherein.

A number of program modules can be stored in the drives and RAM 1112,including an operating system 1130, one or more application programs1132, other program modules 1134 and program data 1136. All or portionsof the operating system, applications, modules, and/or data can also becached in the RAM 1112. The systems and methods described herein can beimplemented utilizing various commercially available operating systemsor combinations of operating systems.

Computer 1102 can optionally comprise emulation technologies. Forexample, a hypervisor (not shown) or other intermediary can emulate ahardware environment for operating system 1130, and the emulatedhardware can optionally be different from the hardware illustrated inFIG. 11. In such an embodiment, operating system 1130 can comprise onevirtual machine (VM) of multiple VMs hosted at computer 1102.Furthermore, operating system 1130 can provide runtime environments,such as the Java runtime environment or the .NET framework, forapplications 1132. Runtime environments are consistent executionenvironments that allow applications 1132 to run on any operating systemthat includes the runtime environment. Similarly, operating system 1130can support containers, and applications 1132 can be in the form ofcontainers, which are lightweight, standalone, executable packages ofsoftware that include, e.g., code, runtime, system tools, systemlibraries and settings for an application.

Further, computer 1102 can be enable with a security module, such as atrusted processing module (TPM). For instance, with a TPM, bootcomponents hash next in time boot components, and wait for a match ofresults to secured values, before loading a next boot component. Thisprocess can take place at any layer in the code execution stack ofcomputer 1102, e.g., applied at the application execution level or atthe operating system (OS) kernel level, thereby enabling security at anylevel of code execution.

A user can enter commands and information into the computer 1102 throughone or more wired/wireless input devices, e.g., a keyboard 1138, a touchscreen 1140, and a pointing device, such as a mouse 1142. Other inputdevices (not shown) can include a microphone, an infrared (IR) remotecontrol, a radio frequency (RF) remote control, or other remote control,a joystick, a virtual reality controller and/or virtual reality headset,a game pad, a stylus pen, an image input device, e.g., camera(s), agesture sensor input device, a vision movement sensor input device, anemotion or facial detection device, a biometric input device, e.g.,fingerprint or iris scanner, or the like. These and other input devicesare often connected to the processing unit 1104 through an input deviceinterface 1144 that can be coupled to the system bus 1108, but can beconnected by other interfaces, such as a parallel port, an IEEE 1394serial port, a game port, a USB port, an IR interface, a BLUETOOTH®interface, etc.

A monitor 1146 or other type of display device can be also connected tothe system bus 1108 via an interface, such as a video adapter 1148. Inaddition to the monitor 1146, a computer typically includes otherperipheral output devices (not shown), such as speakers, printers, etc.

The computer 1102 can operate in a networked environment using logicalconnections via wired and/or wireless communications to one or moreremote computers, such as a remote computer(s) 1150. The remotecomputer(s) 1150 can be a workstation, a server computer, a router, apersonal computer, portable computer, microprocessor-based entertainmentappliance, a peer device or other common network node, and typicallyincludes many or all of the elements described relative to the computer1102, although, for purposes of brevity, only a memory/storage device1152 is illustrated. The logical connections depicted includewired/wireless connectivity to a local area network (LAN) 1154 and/orlarger networks, e.g., a wide area network (WAN) 1156. Such LAN and WANnetworking environments are commonplace in offices and companies, andfacilitate enterprise-wide computer networks, such as intranets, all ofwhich can connect to a global communications network, e.g., theInternet.

When used in a LAN networking environment, the computer 1102 can beconnected to the local network 1154 through a wired and/or wirelesscommunication network interface or adapter 1158. The adapter 1158 canfacilitate wired or wireless communication to the LAN 1154, which canalso include a wireless access point (AP) disposed thereon forcommunicating with the adapter 1158 in a wireless mode.

When used in a WAN networking environment, the computer 1102 can includea modem 1160 or can be connected to a communications server on the WAN1156 via other means for establishing communications over the WAN 1156,such as by way of the Internet. The modem 1160, which can be internal orexternal and a wired or wireless device, can be connected to the systembus 1108 via the input device interface 1144. In a networkedenvironment, program modules depicted relative to the computer 1102 orportions thereof, can be stored in the remote memory/storage device1152. It will be appreciated that the network connections shown areexample and other means of establishing a communications link betweenthe computers can be used.

When used in either a LAN or WAN networking environment, the computer1102 can access cloud storage systems or other network-based storagesystems in addition to, or in place of, external storage devices 1116 asdescribed above. Generally, a connection between the computer 1102 and acloud storage system can be established over a LAN 1154 or WAN 1156e.g., by the adapter 1158 or modem 1160, respectively. Upon connectingthe computer 1102 to an associated cloud storage system, the externalstorage interface 1126 can, with the aid of the adapter 1158 and/ormodem 1160, manage storage provided by the cloud storage system as itwould other types of external storage. For instance, the externalstorage interface 1126 can be configured to provide access to cloudstorage sources as if those sources were physically connected to thecomputer 1102.

The computer 1102 can be operable to communicate with any wirelessdevices or entities operatively disposed in wireless communication,e.g., a printer, scanner, desktop and/or portable computer, portabledata assistant, communications satellite, any piece of equipment orlocation associated with a wirelessly detectable tag (e.g., a kiosk,news stand, store shelf, etc.), and telephone. This can include WirelessFidelity (Wi-Fi) and BLUETOOTH® wireless technologies. Thus, thecommunication can be a predefined structure as with a conventionalnetwork or simply an ad hoc communication between at least two devices.

The above description of illustrated embodiments of the subjectdisclosure, comprising what is described in the Abstract, is notintended to be exhaustive or to limit the disclosed embodiments to theprecise forms disclosed. While specific embodiments and examples aredescribed herein for illustrative purposes, various modifications arepossible that are considered within the scope of such embodiments andexamples, as those skilled in the relevant art can recognize.

In this regard, while the disclosed subject matter has been described inconnection with various embodiments and corresponding Figures, whereapplicable, it is to be understood that other similar embodiments can beused or modifications and additions can be made to the describedembodiments for performing the same, similar, alternative, or substitutefunction of the disclosed subject matter without deviating therefrom.Therefore, the disclosed subject matter should not be limited to anysingle embodiment described herein, but rather should be construed inbreadth and scope in accordance with the appended claims below.

As it employed in the subject specification, the term “processor” canrefer to substantially any computing processing unit or devicecomprising, but not limited to comprising, single-core processors;single-processors with software multithread execution capability;multi-core processors; multi-core processors with software multithreadexecution capability; multi-core processors with hardware multithreadtechnology; parallel platforms; and parallel platforms with distributedshared memory. Additionally, a processor can refer to an integratedcircuit, an application specific integrated circuit, a digital signalprocessor, a field programmable gate array, a programmable logiccontroller, a complex programmable logic device, a discrete gate ortransistor logic, discrete hardware components, or any combinationthereof designed to perform the functions described herein. Processorscan exploit nano-scale architectures such as, but not limited to,molecular and quantum-dot based transistors, switches and gates, inorder to optimize space usage or enhance performance of user equipment.A processor may also be implemented as a combination of computingprocessing units.

As used in this application, the terms “component,” “system,”“platform,” “layer,” “selector,” “interface,” and the like are intendedto refer to a computer-related entity or an entity related to anoperational apparatus with one or more specific functionalities, whereinthe entity can be either hardware, a combination of hardware andsoftware, software, or software in execution. As an example, a componentmay be, but is not limited to being, a process running on a processor, aprocessor, an object, an executable, a thread of execution, a program,and/or a computer. By way of illustration and not limitation, both anapplication running on a server and the server can be a component. Oneor more components may reside within a process and/or thread ofexecution and a component may be localized on one computer and/ordistributed between two or more computers. In addition, these componentscan execute from various computer readable media having various datastructures stored thereon. The components may communicate via localand/or remote processes such as in accordance with a signal having oneor more data packets (e.g., data from one component interacting withanother component in a local system, distributed system, and/or across anetwork such as the Internet with other systems via the signal). Asanother example, a component can be an apparatus with specificfunctionality provided by mechanical parts operated by electric orelectronic circuitry, which is operated by a software or a firmwareapplication executed by a processor, wherein the processor can beinternal or external to the apparatus and executes at least a part ofthe software or firmware application. As yet another example, acomponent can be an apparatus that provides specific functionalitythrough electronic components without mechanical parts, the electroniccomponents can comprise a processor therein to execute software orfirmware that confers at least in part the functionality of theelectronic components.

In addition, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom context, “X employs A or B” is intended to mean any of the naturalinclusive permutations. That is, if X employs A; X employs B; or Xemploys both A and B, then “X employs A or B” is satisfied under any ofthe foregoing instances.

While the embodiments are susceptible to various modifications andalternative constructions, certain illustrated implementations thereofare shown in the drawings and have been described above in detail. Itshould be understood, however, that there is no intention to limit thevarious embodiments to the specific forms disclosed, but on thecontrary, the intention is to cover all modifications, alternativeconstructions, and equivalents falling within the spirit and scope.

In addition to the various implementations described herein, it is to beunderstood that other similar implementations can be used ormodifications and additions can be made to the describedimplementation(s) for performing the same or equivalent function of thecorresponding implementation(s) without deviating therefrom. Stillfurther, multiple processing chips or multiple devices can share theperformance of one or more functions described herein, and similarly,storage can be effected across a plurality of devices. Accordingly, thevarious embodiments are not to be limited to any single implementation,but rather is to be construed in breadth, spirit and scope in accordancewith the appended claims.

What is claimed is:
 1. A system, comprising: a processor; and a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising: traversing a tree data structure to detect an orphaned content addressable storage object comprising a content addressable storage object not referenced by a descriptor file; determining whether the orphaned content addressable storage object satisfies an age criterion; in response to determining that the detected orphaned content addressable storage object satisfies the age criterion, deleting the content addressable storage object resulting in a deleted content addressable storage object; determining whether any new descriptor file written during the traversing has restored the deleted content addressable storage object; and in response to determining that the deleted content addressable storage object has not been restored during the traversing, reclaiming capacity occupied by the deleted content addressable storage object.
 2. The system of claim 1, wherein the operations further comprise creating a snapshot of a bucket containing the content addressable storage object.
 3. The system of claim 2, wherein the operations further comprise deleting the snapshot after determining whether the any new descriptor file written during the traversing has restored the deleted content addressable storage object.
 4. The system of claim 2, wherein the orphaned content addressable storage object is a first orphaned content addressable storage object, and wherein the operations further comprise creating a new descriptor file, determining that a second orphaned content addressable storage object referenced by the new descriptor file does not exist, and restoring the second orphaned content addressable storage object from the snapshot.
 5. The system of claim 1, wherein the age criterion is based on a maximum duration of a descriptor file write transaction.
 6. The system of claim 5, wherein the maximum duration is two weeks.
 7. The system of claim 1, wherein the deleting the content addressable storage object is performed by a first garbage collection process, and wherein the reclaiming the capacity occupied by the deleted content addressable storage object is performed by a second garbage collection engine.
 8. The system of claim 1, wherein the detected content addressable storage object is a binary large object.
 9. The system of claim 1, wherein the descriptor file is a C-Clip descriptor file.
 10. A method, comprising: creating, via a processor of a data storage system, a snapshot of a container containing content addressable storage objects; traversing an object references data structure corresponding to the container to determine orphaned content addressable storage objects comprising content addressable storage objects that are not referenced by at least one descriptor file; deleting, from the container, the orphaned content addressable storage objects that are older than a predetermined descriptor file write duration, the deleting resulting in deleted content addressable storage objects; and reclaiming capacity occupied by the deleted content addressable storage objects that have not been restored from the snapshot via any new descriptor file created during the traversing.
 11. The method of claim 10, wherein the traversing the object references data structure comprises running a first garbage collection operation, and wherein the reclaiming the capacity comprises running a second garbage collection operation.
 12. The method of claim 10, further comprising writing a new descriptor file, during the traversing and before the reclaiming the capacity, that references a deleted content addressable storage object, and restoring the content addressable storage object from the snapshot to an undeleted state.
 13. The method of claim 12, wherein the writing the new descriptor file comprises running a foreground process, and wherein the restoring the content addressable storage object from the snapshot to the undeleted state comprises running a background process.
 14. The method of claim 13, wherein the reclaiming the capacity is performed after the background process completes.
 15. The method of claim 10, wherein the traversing the object references data structure comprises traversing a search tree.
 16. The method of claim 10, wherein the creating the snapshot of the container comprises creating a first snapshot of a first container, and further comprising creating respective one or more snapshots of one or more respective containers.
 17. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor of a data storage system, facilitate performance of operations, the operations comprising: creating a snapshot of a container containing content addressable storage objects; deleting, as part of a first garbage collection process, a content addressable storage object, maintained in the container, that is not referenced by a descriptor file and is older than an age that is based on a defined limit on duration of a descriptor file write transaction; and reclaiming capacity, as part of a second garbage collection process, the content addressable storage object in response to determining that the content addressable storage object was unable to be restored from the snapshot during the first garbage collection process.
 18. The non-transitory machine-readable medium of claim 17, wherein the content addressable storage object is a first content addressable storage object, and wherein the operations further comprise, deleting, as part of the first garbage collection process, a second content addressable storage object, maintained in the container, that is not referenced by any descriptor file and is older than the age that is based on the defined limit on the duration of a descriptor file write transaction, writing a new descriptor file with a reference to the second content addressable storage object, restoring the second content addressable storage object from the snapshot, and avoiding reclamation of capacity of the second content addressable storage object during the second garbage collection process.
 19. The non-transitory machine-readable medium of claim 18, wherein the writing the new descriptor file comprises running a foreground process, wherein the restoring the second content addressable storage object from the snapshot comprises running a background process that starts after the foreground process completes, and wherein the reclaiming the capacity in the second garbage collection process starts after the background process completes.
 20. The non-transitory machine-readable medium of claim 17, wherein the operations comprise traversing an object references data structure as part of the first garbage collection process. 